Method and device for synthesizing image

ABSTRACT

A method and device for synthesizing an image for audiovisual communication includes: generating, if a request for synthesizing a sensor image input through a camera sensor and an image designated by a user is input, a synthesis image of the sensor image and the designated image. The method also includes storing, when the synthesis image is generated, object information of an object of the sensor image and object information of an object of the designated image; and changing, if a sensor image is input in which object information of the sensor image object has changed, object information of the designated image object according to the changed object information of the sensor image object and synthesizing the sensor image and the designated image in which object information has changed.

CROSS-REFERENCE TO RELATED APPLICATION(S) AND CLAIM OF PRIORITY

The present application is related to and claims priority to and thebenefit of Korean Patent Application No. 10-2008-0115019 filed in theKorean Intellectual Property Office on Nov. 19, 2008, the entirecontents of which are incorporated herein by reference.

TECHNICAL FIELD OF THE INVENTION

The present invention relates to image synthesis in a mobile terminal,and more particularly, to a method and device for synthesizing an imagefor audiovisual communication in a mobile terminal.

BACKGROUND OF THE INVENTION

Due to development of technology, a service using an image in a mobileterminal is provided and widely used. For example, a service foracquiring an image using a camera unit provided in a mobile terminal anda service for receiving an image transmitted from another mobileterminal are widely used. Currently, such a service for using an imagein a mobile terminal has developed from a simple use of an image toservices such as editing and synthesis of images.

A second generation mobile communication system provides an audiodedicated communication service, whereas a third generation mobilecommunication system provides a communication service using packet dataand multimedia data as well as audio dedicated communication, therebyproviding a service using an image. The service using an image in amobile terminal may include an audiovisual communication service forperforming communication while viewing another party. The audiovisualcommunication service is performed by simultaneously transmitting imagedata input through a camera unit and audio data input through amicrophone. The audiovisual communication service provides a visualfunction of viewing another party in addition to an audio dedicatedcommunication service through transmission of image data including auser's image, i.e., image transmission. However, because the audiovisualcommunication service provides an image input through a camera unit, aportion of a user's image that the user may be reluctant to expose isprovided to another party, or a particular character of a user that theuser desires to portray may be not shown by simply transmitting animage. In order to solve the problem, a replacement image can beprovided, or image editing such as image synthesis can be used. However,because the audiovisual communication service transmits an image inputthrough a camera sensor in real time, when an image is edited or whenimages are synthesized, changes of an object in real time are notreflected in the transmitted images. Therefore, a method of solving theproblem is requested.

SUMMARY OF THE INVENTION

To address the above-discussed deficiencies of the prior art, it is aprimary object to provide a method and device for synthesizing an image.The present invention further provides a method and device forsynthesizing an image for audiovisual communication.

In accordance with an aspect of the present invention, a method ofsynthesizing an image in a mobile terminal includes: generating, if arequest for synthesizing a sensor image input through a camera sensorand an image designated by a user is input, a synthesis image of thesensor image and the designated image; storing, when the synthesis imageis generated, object information of an object of the sensor image andobject information of an object of the designated image; and changing,if a sensor image is input in which object information of the sensorimage object has changed, object information of the designated imageobject according to the changed object information of the sensor imageobject and synthesizing the sensor image and the designated image inwhich object information has changed.

In accordance with another aspect of the present invention, an imagesynthesizing device in a mobile terminal includes: a camera unitconfigured to input a sensor image; a display unit configured to displaythe sensor image and an image designated by a user; a controllerconfigured to synthesize, if a request for synthesizing an image isinput, the sensor image displayed in the display unit and the designatedimage and change, if object information of an object of the sensor imagechanges, object information of an object of the designated image tocorrespond to the change of object information of the sensor imageobject and further configured to synthesize the sensor image and thedesignated image in which object information has changed; and a storageunit configured to store, when the images are synthesized, the objectinformation of the sensor image object and the object information of thedesignated image object.

The object of the present invention is not limited to theabove-described objects and other objects will be understood by thoseskilled in the art from the following description.

Before undertaking the DETAILED DESCRIPTION OF THE INVENTION below, itmay be advantageous to set forth definitions of certain words andphrases used throughout this patent document: the terms “include” and“comprise,” as well as derivatives thereof, mean inclusion withoutlimitation; the term “or,” is inclusive, meaning and/or; the phrases“associated with” and “associated therewith,” as well as derivativesthereof, may mean to include, be included within, interconnect with,contain, be contained within, connect to or with, couple to or with, becommunicable with, cooperate with, interleave, juxtapose, be proximateto, be bound to or with, have, have a property of, or the like; and theterm “controller” means any device, system or part thereof that controlsat least one operation, such a device may be implemented in hardware,firmware or software, or some combination of at least two of the same.It should be noted that the functionality associated with any particularcontroller may be centralized or distributed, whether locally orremotely. Definitions for certain words and phrases are providedthroughout this patent document, those of ordinary skill in the artshould understand that in many, if not most instances, such definitionsapply to prior, as well as future uses of such defined words andphrases.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present disclosure and itsadvantages, reference is now made to the following description taken inconjunction with the accompanying drawings, in which like referencenumerals represent like parts:

FIG. 1 illustrates a block diagram of a configuration of a mobileterminal for audiovisual communication according to an exemplaryembodiment of the present invention;

FIG. 2A illustrates screen examples for a method of synthesizing animage for audiovisual communication according to another exemplaryembodiment of the present invention;

FIG. 2B illustrates screen examples for a change of a synthesis imagewhen an object of a sensor image moves while performing audiovisualcommunication in the method of FIG. 2A;

FIG. 3A illustrates screen examples for a method of synthesizing animage for audiovisual communication according to another exemplaryembodiment of the present invention;

FIG. 3B illustrates screen examples for a change of a synthesis imagewhen an object of a sensor image moves while performing audiovisualcommunication in the method of FIG. 3A;

FIG. 4 illustrates a flowchart for a method of performing audiovisualcommunication using a synthesis image according to another exemplaryembodiment of the present invention; and

FIG. 5 illustrates a flowchart for a method of performing audiovisualcommunication using a synthesis image according to another exemplaryembodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIGS. 1 through 5, discussed below, and the various embodiments used todescribe the principles of the present disclosure in this patentdocument are by way of illustration only and should not be construed inany way to limit the scope of the disclosure. Those skilled in the artwill understand that the principles of the present disclosure may beimplemented in any suitably arranged mobile terminal.

In the following description, a case of synthesizing an image providedto another party's terminal while performing audiovisual communicationis exemplified. However, the present invention is not limited tosynthesis of an image when performing audiovisual communication and canbe applied to a case of synthesizing a still image in which an object ofthe image does not move.

FIG. 1 illustrates a block diagram for a configuration of a mobileterminal for audiovisual communication according to an exemplaryembodiment of the present invention. Although FIG. 1 illustrates amobile communication terminal for performing audiovisual communication,the present invention is not limited thereto and can be applied to anyterminal that includes a camera unit and capable of performingaudiovisual communication.

Referring to FIG. 1, a mobile terminal capable of audiovisualcommunication according to the present exemplary embodiment includes aradio frequency (RF) unit 110, audio processor 120, controller 130,storage unit 140, camera unit 150, image processor 160, input unit 170,and display unit 180. The RF unit 110 transmits and receives a wirelesssignal to perform wireless communication with a base station. For this,the RF unit 110 up-converts a base-band signal into an RF signal andamplifies the RF signal, and low-noise amplifies an RF signal receivedthrough an antenna and down-converts the low-noise amplified RF signalinto an base-band signal, and outputs the down-converted signal to thecontroller 130. The RF unit 110 outputs an audiovisual communicationsignal received through the antenna to the controller 130 and transmitsan input audiovisual communication signal received from the controller130 to a base station through the antenna. The audio processor 120processes the input audio signal. Specifically, the audio processor 120decodes an audio signal received from the controller 130 and outputs theaudio signal through a speaker SPK, and converts an audio signal inputthrough a microphone MIC to a digital signal and outputs the converteddigital signal to the controller 130. The controller 130 controlsgeneral operations of the mobile terminal. Particularly, if a requestfor image synthesis for audiovisual communication is input, thecontroller 130 synthesizes an image input through the camera unit 150and an image designated by a user. The controller also encodes thesynthesized image into an image data format for audiovisualcommunication, and transmits the encoded image data format to anotherparty. The storage unit 140 stores programs and data for performingvarious functions of the mobile terminal. Further, the storage unit 140stores still images in units of a frame and stores, when synthesizingimages, position information of an object of a sensor image inputthrough a camera sensor of the camera unit 150 and position informationof an object of a designated image.

The camera unit 150 receives an image signal to photograph an externalobject. In order to input an image signal, the camera unit 150 includesa camera sensor that converts the input optical signal to an analogelectrical signal and a signal processor that converts the analog signalconverted by the camera sensor to a digital signal. The image processor160 processes an image signal output by the camera unit 150 tocorrespond to a characteristic and a size of the display unit 180 andoutputs the image signal. An image codec that compresses an image signalin a preset method and that restores a compressed image signal to anoriginal image signal is included in the image processor 160 or thecontroller 130. The image codec may be a Joint Photographic CodingExperts Group (JPEG) codec or a Moving Picture Experts Group (MPEG) 4codec. When images are synthesized to perform audiovisual communication,the synthesized images are encoded into a data format for audiovisualcommunication through the image codec. Further, the image processor 160may have a frame buffer (not shown) that stores a frame image data inputthrough the camera unit 150 and processed in units of a frame. When arequest for forming an image is input, a formed image is converted to aframe and synthesized with frame image data stored in the frame buffer.

The input unit 170 can be formed as a keypad or a touch pad. The inputunit 170 includes character keys, numeral keys, various function keys,and an external volume key. The input unit 170 outputs a key inputsignal corresponding to a key input by a user to the controller 130. Thedisplay unit 180 displays an image signal output by the image processor160 on a screen and displays user data output by the controller 130 onthe screen. The display unit 180 can use a liquid crystal display (LCD),including an LCD controller, memory unit that stores image data, and LCDdisplay element. When the LCD uses a touch screen method, the LCD alsoperforms a function of an input unit, as described in detail later inrelation to FIGS. 3A, 3B, and 5.

FIG. 2A shows screen examples for a method of synthesizing an image foraudiovisual communication according to another exemplary embodiment ofthe present invention. The screen examples illustrate a case forsynthesizing an input sensor image and a stored image before performingaudiovisual communication.

Referring to FIG. 2A, in a screen 210, a user inputs another party'sinformation, such as a phone number, in order to start communication.Upon inputting another party's information, a screen 220 is displayed,and the user selects a kind of communication to perform using the inputinformation of the other party. A communication kind list 221 isdisplayed in the screen 220 to select the kind of communication. Here, acommunication request method is not limited to the above-describedmethod. For example, a communication request may be directly performedwithout a process of selecting a communication kind by inputting anotherparty's information to the screen 210 and then inputting a specific keydedicated to audio communication or to audiovisual communication in theinput unit 170. In the illustrated example, the user selects audiovisualcommunication from the communication kind list 221 of the screen 220,and a list 231 of audiovisual communication functions is displayed in ascreen 230. The user selects a function for a stored image selectionfrom the list 231 of audiovisual communication functions of the screen230. Here, the function of stored image selection is a function ofselecting a stored image for synthesizing with a sensor image inputthrough the camera sensor of the camera unit 150 (hereinafter, a ‘sensorimage’). Upon selection of the stored image selection function, a list241 of images is displayed in the screen 240 in order to select one ofthe stored images. In the example, a folder including a list of storedimages is displayed in the list 241 as an ‘image storage box’. Thedisplayed folder can be changed according to user selection. Further, inthe list 231 of audiovisual communication functions displayed in thescreen 230, an image forming function is a function of forming an imageto be synthesized with a sensor image and is described in detail laterin relation to FIG. 3A.

In the example, an image file ‘partition.jpg’ is selected from the list241 of the stored images in the screen 240, and the stored image‘partition.jpg’ is displayed in a screen 250 as a stored image 251. In astate where the stored image 251 is displayed, an object of a sensorimage 261, such as a user's face, is displayed in a screen 260. In thiscase, the sensor image 261 and the stored image 251 are overlaid anddisplayed using different layers. The user can choose a position of anobject of the sensor image 261 and a position of an object of the storedimage 251 in the screen 260. Upon completion of positioning the objects,the user inputs a predetermined key notifying completion, and a screen270 is displayed for the user to select whether to generate a synthesisimage for audiovisual communication. If the user inputs a request forgenerating a synthesis image for audiovisual communication to the screen270, the generated synthesis image is displayed in the screen 280. Whengenerating a synthesis image, object information of an object of thestored image and object information of an object of the sensor image arestored in the storage unit 140. Here, the object information of theobject of the stored image and the object information of the object ofthe sensor image are divided and stored separately. In this case, objectinformation of an image includes information representing an object ofan image, such as a size and a position of the object. When transmittingobject information of the synthesis image, it is assumed that an MPEG4encoding method is used for compressing and encoding only changeinformation of the object. When a synthesis image for audiovisualcommunication is generated, the user determines whether to performaudiovisual communication with another party corresponding to anotherparty's information input on the screen 210. If the user inputs arequest for performing audiovisual communication, audiovisualcommunication is performed while transmitting the generated synthesisimage to the other party's terminal. Object information of the storedimage also is changed according to a change of object information of theinput sensor image while performing audiovisual communication, forexample, when an object of the sensor image moves. The stored image issynthesized with the sensor image, and the synthesized image istransmitted to the other party's terminal. This is described in detailwith reference to FIG. 2B.

FIG. 2B shows screen examples illustrating a change of a synthesis imagewhen an object of a sensor image moves while performing audiovisualcommunication in the method of FIG. 2A.

Referring to FIG. 2B, in a screen 290, the synthesis image of the storedimage and the sensor image is displayed in the other party's terminalwhen performing audiovisual communication. An image transmitted toanother party's terminal in order to perform audiovisual communicationis encoded and transmitted by a video encoding method, such as an MPEG4encoding method, for easy transmission. The MPEG4 encoding method is amethod of encoding moving picture frames into an intra-coded frame(I-frame), predictive-coded frame (P-frame), and bidirectional-codedframe (B-frame). The I-frame is a key frame including actual image data,used for random access of data, and is encoded without referring toother images. The P-frame is a frame formed with change informationbased on I-frame information and P-frame information of the previousframe. The B-frame is a frame formed based on I-frame information andP-frame information of the previous frame, and I-frame information andP-frame information of the subsequent frame, and has a much highercompression ratio than that of the I-frame and the P-frame. Accordingly,the B-frame has a lowest size and picture quality. If images aresynthesized as an image for audiovisual communication, as shown in thescreen 280 of FIG. 2A, the synthesized image is stored in a buffer withthe I-frame, and when audiovisual communication is started, thesynthesized image is provided to another party. In the MPEG4 encodingmethod, significant objects are divided from each image using a contoursearch algorithm and compressing and encoding information, such as amotion variable, contour line variable, and color variable of dividedobjects. That is, the MPEG4 encoding method encodes only changeinformation of a changing object without encoding of entire imageinformation. Therefore, when images are synthesized and encoded,information of each divided object is encoded and transmitted to anotherparty's terminal, as shown in the screen 290. After transmitting thescreen 290, if a changed frame corresponding to a screen 295 istransmitted, changed object information between the objects of thescreen 290 and the screen 295 is encoded and transmitted. A process ofsynthesizing an image after an object change of the stored imageaccording to an object change of the sensor image, such as when thesensor image moves, as in the screen 295, is described later withreference to FIGS. 4 and 5.

In a state where a synthesis image of the sensor image and the storedimage is displayed, as in the screen 290, when an object 291 of thesensor image moves, a changed image is displayed in the screen 295. Whenposition information of the object 291 of the sensor image of the screen290 changes, such as when the object 291 moves, a new object 296 of asensor image is displayed in the screen 295, and a position of theobject 292 of the stored image on the screen 290 is moved accordingly toa new position of an object 297 displayed in the screen 295 by using thechanged object information of the object 292 of the stored image in thescreen 290. The screen 295 of FIG. 2B illustrates a example where aposition of the object on the screen 290 changes from a previous frame,such as a case where the object on the screen 290 moves, however, anobject of the stored image also can be changed according to an objectchange of the sensor image when another information of an objectchanges, such as when a size of the object on the screen 290 changes. Asshown in the screen 295, if the object 296 of the sensor image moves,such as if another party's face moves, the object 297 of a stored image,such as a partition.jpg, also is moved and thus, a synthesized image isdisplayed. Accordingly, in the present exemplary embodiment, whenperforming audiovisual communication, an object of a stored image alsois changed according to a change of an object of a sensor image, such asmovement or a size change of an object of the sensor image, and thus, animage that the user desires to transmit can be maintained untilaudiovisual communication terminates.

FIG. 3A shows screen examples illustrating a method of synthesizing animage for audiovisual communication according to another exemplaryembodiment of the present invention. The screen examples illustrate acase of synthesizing an input sensor image and an image formed whileperforming audiovisual communication, using a touch screen of thedisplay unit 180 to perform a function of an input unit.

Referring to FIG. 3A, a screen 310 is displayed in the display unit 180when performing audiovisual communication. An image 311 of another partytransmitted from the other party's terminal for audiovisualcommunication and a sensor image 312 are each displayed in apredetermined area in the screen 310. The user performs a predeterminedinput method to display a list 321 of functions related to audiovisualcommunication in a screen 320, such as by inputting a ‘menu’ keyprovided in the input unit 170, while performing audiovisualcommunication in a state of the screen 310. If the user selects an itemof ‘forming of a synthesis image’ from the list 321 of functions relatedto audiovisual communication, a screen 320 is displayed, in which theuser can interchange areas of the display unit 180, in which a sensorimage 331 and another party's image 332 are displayed when performingaudiovisual communication. That is, a display area of the sensor image312 displayed in the screen 310 can be changed to a display area of thesensor image 331 displayed in the screen 330, and a display area ofanother party's image 311 displayed in the screen 310 can be changed toa display area of another party's image 332 displayed in the screen 330.Further, when the screen 310 is converted to the screen 330 in order toform an image, an image forming tool 332, such as an image forming pen,may be displayed in the screen 330. In FIG. 3A, the display unit 180includes a touch screen providing a function of the input unit 170,wherein a touch sensor detects a user's operation in the display unit180. The user's operation is detected by detecting a touch of thedisplay unit 180 by the user, and an image can be formed through thedetected operation. However, other methods of detecting a user'soperation in the display unit 180 may be used. For example, the mobileterminal may have a heat sensor for detecting heat emitted towards thedisplay unit 180 by the user or a pressure sensor for detecting apressure level of the display unit 180 in addition to a touch sensor.

An image is formed using the image forming tool 333 displayed in thescreen 330. Other image forming means may be used in place of thedisplayed image forming tool 333, for example touching the touch screenby a finger or touching tool. Further, the user can select various typesof input using the image forming tool 333, for example a thickness,shape, and color of a line. A screen 340 shows a screen example in whichan image 344 (hereinafter, a ‘formed image’) is formed using the imageforming tool 333 in the display area of the display unit 180 thatdisplays the sensor image 331. When forming of the formed image 344 iscomplete, the user inputs a predetermined key notifying completion ofimage forming in the screen 340, such as a completion key 345, and thescreen 340 is converted to a screen 350.

The formed image 344 generated on the screen 340 is stored as a formedimage. The formed image 344 and the sensor image 341 are synthesized inthe screen 350. Object information of an object of the synthesizedformed image and object information of an object of the sensor image arestored in the storage unit 140. When forming a synthesis image, it isassumed that the object information of the object of the stored imageand the object information of the object of the sensor image are dividedand stored separately. In this case, object information of an imageincludes any information representing an object of the image, such as asize and a position of an object. The formed image 344 can be generatedusing various methods, such as, a method for detecting a user'soperation through the input unit 170, acquiring multiple coordinates ofobjects of a formed image, and thereby acquiring position information ofa point, line, and surface for forming the object. In this case,position information of an object is obtained by storing inputinformation as a layer in a buffer whenever position information inputby a user constitutes a unit. When an input notifying completion ofimage forming by a user is received, a formed image can be generated bysynthesizing the stored layers. Further, position informationconstituting a unit may be an input of a single position at more thanone time, an input of a plurality of positions at more than one time,and position movement of an input state at more than one time within apredetermined time period. When image synthesis is completed, thedisplay area of the sensor image 331 and the display area of anotherparty's image 332 displayed in the screen 340 are again interchanged,and a screen 360 displays a sensor image 362 and another party's image361. A state of the mobile terminal is returned to the state existingbefore forming of the synthesized image, and thus, audiovisualcommunication can be performed continuously. In this case, the sensorimage 362 is the synthesized image formed by synthesizing the sensorimage 331 and the formed image 344 generated in the screen 340.

FIG. 3B shows screen examples for a change of a synthesis image when anobject of a sensor image moves while performing audiovisualcommunication in the method of FIG. 3A.

Referring to FIG. 3B, a screen 370 is displayed in another party'sterminal for performing audiovisual communication using a synthesisimage of a sensor image and a formed image, as shown in FIG. 3A.Further, a screen 380 is a screen example in which a further imagesignal is received and decoded after displaying the screen 370, and inwhich the image signal is displayed in another party's terminal in orderto perform audiovisual communication. FIG. 3B illustrates screenexamples displayed in another party's terminal in order to illustrate anexample of audiovisual communication. For convenience of description,the following description is described based on a mobile terminal thatcan perform image synthesis. Accordingly, a synthesis image synthesizedfrom a previously input sensor image, a formed image, and an image 372input through a camera unit of another party's terminal, are displayedin the screen 370. Further, a synthesis image of a presently inputsensor image and a formed image can be displayed, as in a screen 380.When position information of an object 371 of the sensor image of thescreen 370 changes, such as when the object 371 moves, an object 373 ofa sensor image of the screen 370 is moved to an object 382 of a formedimage of the screen 380 by changing object information of an object 373of a formed image of the screen 370 using the changed positioninformation. A process of changing an object of a formed image as anobject of the sensor image changes, and of synthesizing an image therefrom, as in the screen 380, is described later with reference to FIGS. 4and 5. The screen 380 of FIG. 3B illustrates a case where a position ofan object on the screen 370 changes from a previous frame, such as acase where an object moves, however an object of a formed image can alsobe changed according to an object change of the sensor image whenanother information of an object of the sensor image changes, such aswhen a size of the object changes. Accordingly, in the present exemplaryembodiment, when performing audiovisual communication, an object of astored image also is changed and displayed according to an object changeof the sensor image, such as movement or a change of a size of an objectof a sensor image. Therefore, an image intended by a user can bemaintained until audiovisual communication terminates.

In the foregoing description, a case of synthesizing an image from astored image and a sensor image before performing audiovisualcommunication and a case of synthesizing an image from a formed imageand a sensor image while performing audiovisual communication aredescribed. However, an image can be synthesized in other cases. Forexample, an image can be synthesized from a formed image and a sensorimage before performing audiovisual communication, or an image can besynthesized from a stored image and a sensor image while performingaudiovisual communication.

FIG. 4 illustrates a flowchart a method of performing audiovisualcommunication using a synthesis image in a mobile terminal according toanother exemplary embodiment of the present invention. FIG. 4illustrates a case of synthesizing an image before performing theaudiovisual communication.

Referring to FIG. 4, when the mobile terminal is in an audiovisualcommunication standby mode (block 405), the controller 130 determineswhether a request for synthesis of a stored image is input (block 410).Here, the audiovisual communication standby mode is a state in whichanother party's information is input and an audiovisual communicationfunction can be selected, as in the screen 220 of FIG. 2A. If a requestfor synthesis of a stored image is input, the controller 130 receives aselection input of a stored image to be synthesized (block 415). In thiscase, in order for the user to select an image to be synthesized, a listof stored images is displayed in the display unit 180. The controller130 controls the display unit 180 to display the selected stored imageand a sensor image input through the camera unit 150, as in the screen260 (block 420). If a request for synthesis of the selected stored imageand the sensor image is input, the controller 130 synthesizes the storedimage and the sensor image displayed in the display unit 180 into asynthesis image (block 425). Here, the sensor image is an imageprocessed by the image processor 160 in units of a frame. The imageprocessor 160 receives image signals input through the camera sensor ofthe camera unit 150 until the input image signals form a frame unit,then processes the image signal frame unit according to a size andcharacteristic of the display unit 180 and displays the resulting image.The sensor image to be synthesized with the stored image is an image ofa frame unit, and the stored image is also stored in units of a frame.

The controller 130 stores object information of objects of the synthesisimage (block 430). For example, the controller 130 controls the storageunit 140 to store object information of the sensor image, such asanother party's face, and object information of the stored image, suchas a position and a color of an object divided by a partition, as in thescreen 280. The synthesis image is stored in a buffer of the storageunit 140 or of the image processor 160. Particularly, in order to enablechange of object information of a stored synthesis image according to achange of object information of the sensor image, object information ofthe synthesis image is stored in the storage unit 140. The controller130 determines whether a request for audiovisual communication is input(block 435). If a request for audiovisual communication is input, thecontroller 130 transmits the synthesis image as an audiovisualcommunication image to another party's terminal (block 440). In order toperform audiovisual communication, the sensor image is continuouslytransmitted to another party's terminal in units of a frame. Thecontroller 130 continues to receive an input sensor image through thecamera unit 150. When a sensor image corresponding to a quantity of aframe is input, the controller 130 compares object information of apresent frame with that of a previous frame, and thereby determineswhether object information of the sensor image changes (block 445).Particularly, the controller 130 determines whether object informationof an object synthesized with an image that is designated by a user(hereinafter, a ‘designated image’), for example the object 291 of FIG.2B and the object 371 of FIG. 3B, changes. For convenience ofdescription, a frame image in which a sensor image is presently input inunits of a frame through the camera unit 150 is referred to as a‘present sensor image’, and a sensor image of the frame immediatelypreceding the present input frame is referred to as a ‘previous sensorimage’. When information of an object of the sensor image changes, thecontroller 130 changes object information of a designated image, such asa stored image or a formed image to be synthesized with the presentsensor image, synthesizes the images, encodes the synthesis image, andtransmits the synthesis image to another party's terminal according tochange information between objects of the present sensor image and theprevious sensor image (block 450). Here, an encoded image is informationof only the changed objects when comparing with the previous image, asdescribed herein above, and only the changed information of objects isencoded and transmitted to another party's terminal. The changeinformation includes information such as a changed position, size, androtational orientation of an object. Change information of a positioncan be acquired by obtaining a motion variable, for example a motionvector, between compared objects. A method of obtaining a motion vectorcan be obtained by a well-known technology and therefore a detaileddescription thereof is omitted. When size information of an objectchanges, information of a size change between objects can be acquiredusing a contour line variable, and an object size of a designated imageis also changed by a change ratio of an object size of the sensor imageaccording to the acquired size change information. Further, when arotational orientation of an object of the sensor image changes,information of the rotational orientation change can be acquired. Changeinformation of a rotational orientation can be obtained by setting acenter of a specific object as a reference orientation axis andmeasuring the change of orientation of the reference orientation axis.Thereafter, a rotational orientation of an object of the designatedimage is also changed according to the obtained change information of arotational orientation of the sensor image. The controller 130 changesobject information of a designated image according to a change of objectinformation of the sensor image using change information acquiredthrough such a method. If no object information of the sensor imagechanges at block 445, the controller 130 synthesizes and encodes thedesignated image and the sensor image without changing objectinformation of the designated image and transmits the encoded signalimage to another party's terminal (block 495). In this case, because noobject information changes, transmission of the image may be omitted.The controller 130 determines whether audiovisual communication isterminated (block 455). If audiovisual communication is not terminated,the process returns to block 445.

If a request for audiovisual communication is not input at block 435,the controller 130 controls the storage unit 140 to store thesynthesized image (block 460). When the synthesis image is stored,information of divided objects of the synthesized image is also stored.The controller 130 determines whether audiovisual communication standbymode is terminated (block 465). If the audiovisual communication standbymode is not terminated, the process returns to block 405.

If a request for synthesis of a stored image is not input at block 410,the controller 130 determines whether a request for forming a synthesisimage is input (block 470). If a request for forming a synthesis imageis not input, the process returns to block 405. If a request for formingthe synthesis image is input, the controller 130 converts a mode of themobile terminal to an image forming mode (block 475) and receives animage forming input (block 480). In this exemplary embodiment, the imageforming mode is a mode of displaying the image forming pen 333 anddetecting an image forming input from the user using the image formingpen 333, as in the screen 330 of FIG. 3A. The image forming input can bedetected by a touch sensor of the input unit 170, and may further bedetected by at least one of a heat sensor and a pressure sensor as wellas the touch sensor. In a mode of detecting an image forming input, whenthe touch sensor of the input unit 170 detects a touch of the displayunit 180 by a user, an image is formed according to the detected touch.When an input notifying completion of image forming is received, thecontroller 130 synthesizes the formed image and a sensor image displayedin the display unit 180 (block 485), and the process then advances toblock 430 for storing object information of objects of the synthesisimage. In the foregoing description, an object of the designated imageis also changed according to an object change of the sensor image.However, the user can select to maintain an object of the designatedimage regardless of an object change of the sensor image. In this case,an enable function of changing a designated image according to a changeof a sensor image and a disable function of not changing a designatedimage regardless of a change of a sensor image are provided to beselected by a user. If the enable function is selected, an object of thedesignated image is also changed according to change information of anobject of the sensor image, as described in FIG. 4, and the changedsensor image and the changed designated image are synthesized. If thedisable function is selected, even if an object of the sensor imagechanges, an object of the designated image is not changed, and thechanged sensor image and the unchanged designated image are synthesized.

Accordingly, if an object of the sensor image changes, an object of astored image or of a formed image to be synthesized with the sensorimage is also changed. Thus, the present invention continues to performaudiovisual communication in a state in which synthesis of images isrequested, such as a synthesis state of objects of an image.

FIG. 5 illustrates a flowchart for a method of performing audiovisualcommunication using a synthesis image according to another exemplaryembodiment of the present invention. FIG. 5 illustrates a case ofsynthesizing an image while performing the audiovisual communication.

Referring to FIG. 5, the controller 130 performs audiovisualcommunication (block 505). The controller 130 determines whether arequest for synthesis of a stored image and a sensor image is input(block 510). If a request for synthesis of a stored image and a sensorimage is input, the controller 130 receives a selection input of astored image to be synthesized (block 515). In this case, in order forthe user to select a stored image to be synthesized, the controller 130controls the display unit 180 to display a list of stored images. Thecontroller 130 controls the display unit 180 to display the selectedstored image and a sensor image input through the camera unit 150, as inthe screen 260 (block 520). If a request for synthesis of the selectedstored image and the sensor image is input, the controller 130synthesizes the stored image and the sensor image displayed in thedisplay unit 180 into a synthesis image (block 525). Here, the sensorimage is an image processed by the image processor 160 in units of aframe. The image processor 160 receives image signals input through thecamera sensor of the camera unit 150 until the input image signals froma frame unit, then processes the image signal frame unit according to asize and characteristic of the display unit 180 and displays theresulting image. The sensor image to be synthesized with the storedimage is an image of a frame unit, and the stored image is also storedin units of a frame.

The controller 130 stores object information of objects of the synthesisimage (block 530). For example, the controller 130 controls the storageunit 140 to store object information of the sensor image, such asanother party's face, and object information of the stored image, suchas a position and a color of an object divided by a partition, as in thescreen 280. The synthesis image is stored in a buffer of the storageunit 140 or of the image processor 160. Particularly, in order to enablechange of object information of a stored synthesis image according to achange of object information of the sensor image, object information ofthe synthesis image is stored in the storage unit 140. Here, objectinformation of objects of a synthesis image includes information of asize, position, and rotational orientation of each object. When some ofthe object information changes, particularly when object information ofthe sensor image changes, object information such as the position, size,and rotational orientation of the designated image is changed accordingto the change information. The controller 130 transmits the synthesisimage as an audiovisual communication image to another party's terminal(block 535). In order to perform audiovisual communication, the sensorimage is continuously transmitted to another party's terminal in unitsof a frame. As described above, the sensor image continuouslytransmitted to another party's terminal is a synthesis image of thedesignated image and the sensor image, and may be a frame including onlychange information of objects divided from the synthesis image by anMPEG4 encoding method. The controller 130 continues to receive an inputsensor image through the camera unit 150. When a sensor imagecorresponding to a quantity of a frame is input, the controller 130compares object information of a present frame with that of a previousframe, and thereby determines whether object information of the sensorimage changes (block 540). Particularly, the controller 130 determineswhether object information of an object synthesized with an object ofthe designated image, such as object information of the object 291 ofFIG. 2B and the object 371 of FIG. 3B, changes. For convenience ofdescription, a frame image in which a sensor image is presently input inunits of a frame through the camera unit 150 is referred to as a‘present sensor image’, and an image of the previous frame immediatelypreceding the present input frame is referred to as a ‘previous sensorimage’. When information of an object of the sensor image changes, thecontroller 130 changes object information of a designated image, such asa stored image or a formed image to be synthesized with the presentsensor image, synthesizes the images, encodes the synthesis image, andtransmits the encoded synthesis image signals to another party'sterminal according to change information between objects of the presentsensor image and the previous sensor image (block 545). Here, an encodedimage is information of only the changed objects when comparing with theprevious synthesis image, as described above. Additionally only thechanged information of objects is encoded and transmitted to anotherparty's terminal. The change information includes information such as achanged position, size, and rotation direction of an object. Changeinformation of a position can be obtained by obtaining a motionvariable, for example a motion vector, between compared objects. Amethod of obtaining a motion vector can be obtained by a well-knowntechnology and therefore a detailed description thereof is omitted. Whensize information of an object changes, information of a size changebetween objects can be acquired using a contour line variable. Further,when a rotational orientation of an object of the sensor image changes,information of the rotational orientation change can be acquired. Changeinformation of a rotational orientation can be obtained by setting acenter of a specific object as a reference orientation axis andmeasuring the change of orientation of the reference orientation axis.The controller 130 changes object information of a designated imageaccording to a change of object information of the sensor image usingchange information acquired through such a method. If no objectinformation of the sensor image changes at step 540, the controller 130synthesizes and encodes the designated image and the sensor imagewithout changing object information of the designated image andtransmits the encoded image signal to another party's terminal (block555). In this case, because no object information changes, transmissionof the image signal may be omitted. The controller 130 determineswhether audiovisual communication is terminated (block 550). Ifaudiovisual communication is not terminated, the process returns to step540.

If a request for synthesis of a stored image is not input at block 510,the controller 130 determines whether a request for forming a synthesisimage is input (block 560). If a request for forming a synthesis imageis not input, the process returns to block 505. If a request for forminga synthesis image is input, the controller 130 converts a mode of themobile terminal to an image forming mode (block 565) and receives animage forming input (block 570). In this exemplary embodiment, the imageforming mode is a mode of displaying the image forming pen 333 anddetecting an image forming input from the user using the image formingpen 333, as in the screen 330 of FIG. 3A. The image forming input can bedetected by a touch sensor of the input unit 170, and may further bedetected by at least one of a heat sensor and a pressure sensor as wellas the touch sensor. When an input notifying completion of image formingis received, the controller 130 synthesizes the formed image and asensor image displayed in the display unit 180 (block 575). The processthen advances to block 530 for storing object information of objects ofthe synthesis image.

As described above, according to the present invention, by performingaudiovisual communication using an image synthesized before or whileperforming audiovisual communication, an undesired exposure of a portionof a user's image can be intercepted, thereby improving a servicequality of audiovisual communication and user satisfaction. Further,when performing audiovisual communication, a user's image to be providedto another party's terminal can be formed with a desired image and canbe provided to another party's terminal, thereby protecting the user'sprivacy and improving user satisfaction. Further, as the mobile terminalhas a function of providing an entertainment device, the presentinvention enables a user to provide various image changes to provideentertainment beyond simple audiovisual communication. Further, in thepresent invention, when an object of an image input through a camerasensor moves, an object of a synthesis image also moves and thus a stateof an image formed by a user, such as a synthesized state of objects ofan image, can be maintained.

Although the present disclosure has been described with an exemplaryembodiment, various changes and modifications may be suggested to oneskilled in the art. It is intended that the present disclosure encompasssuch changes and modifications as fall within the scope of the appendedclaims.

What is claimed is:
 1. A method of synthesizing an image in a mobileterminal, comprising: generating, when a request for synthesizing asensor image input through a camera sensor and an image designated by auser is input, a synthesis image of the sensor image and the designatedimage; storing, when the synthesis image is generated, objectinformation of an object of the sensor image and object information ofan object of the designated image; determining whether objectinformation of the object of a subsequent sensor image input through thecamera sensor is changed; changing, when object information of theobject of the subsequent sensor image input through the camera sensor ischanged, object information of the designated object according to thechanged object information of the object of the sensor image; andsynthesizing the subsequent sensor image and the designated image inwhich object information has changed.
 2. The method of claim 1, whereindetermining whether object information of the object of a subsequentsensor image input through the camera sensor is changed comprises:receiving the subsequent sensor image input through the camera sensor;comparing object information of the object of the subsequent sensorimage of the input sensor image with object information of the object ofthe sensor image of the previously input sensor image; and acquiring,when the object information of the subsequent sensor image and that ofthe previously input sensor image are not identical, the changed objectinformation of the object of the subsequent sensor image.
 3. The methodof claim 2, wherein the change of object information of the object ofthe subsequent sensor image comprises a change of at least one of asize, position, and rotational orientation of the object of the sensorimage.
 4. The method of claim 3, wherein changing object information ofthe designated object comprises changing, when the change of objectinformation of the object of the subsequent sensor image is a change ofposition, the position of the designated object to correspond to thechange of position of the object of the sensor image.
 5. The method ofclaim 3, wherein changing object information of the designated objectcomprises changing, when the change of object information of thesubsequent sensor object is the change of the size, a size of the objectof the designated image by a size change ratio identical to the sizechange ratio of the object of the sensor image.
 6. The method of claim3, wherein changing object information of the designated objectcomprises changing, when the change of object information of the objectof the subsequent sensor image is the change of rotational orientation,a rotational orientation of the object of the designated image tocorrespond to the change of rotational orientation of the object of thesensor image.
 7. The method of claim 1, wherein the generated synthesisimage and the changed synthesis image are transmitted to another party'sterminal when performing audiovisual communication.
 8. The method ofclaim 7, wherein the synthesis images are encoded using MPEG4 fortransmission.
 9. The method of claim 1, further comprising: selecting astored image as the designated image; and displaying the selected storedimage and the sensor image.
 10. The method of claim 1, furthercomprising: converting, when an image is to be formed for designating asthe designated image, a mode of the mobile terminal to an image formingmode; detecting an image input by a user in the image forming mode; andgenerating a formed image according to the detected image input.
 11. Themethod of claim 10, wherein the image input by a user is detected by atouch sensor.
 12. An image synthesizing device in a mobile terminal,comprising: a camera unit configured to input a sensor image; a displayunit configured to display the sensor image and an image designated by auser; a controller configured to synthesize, when a request forsynthesizing an image is input, the sensor image displayed in thedisplay unit and the designated image, the controller further configuredto determine whether object information of an object of a subsequentsensor image input through the camera sensor is changed, change, whenobject information of the object of the subsequent sensor image ischanged, object information of an object of the designated image tocorrespond to the change of object information of the object of thesensor image and synthesize the subsequent sensor image and thedesignated image in which object information has changed; and a storageunit configured to store, when the images are synthesized, the objectinformation of the object of the sensor image and the object informationof the designated object.
 13. The image synthesizing device of claim 12,wherein the controller is configured to compare, when the subsequentsensor image is input through the camera unit, object information of thesensor object of the input subsequent sensor image with objectinformation of the sensor object of a previously input sensor image, andacquire, when the object information of the input subsequent sensorimage and that of the previously input sensor image are not identical,the changed object information of the object of the input subsequentsensor image.
 14. The image synthesizing device of claim 13, wherein thechange of object information of the sensor object comprises a change ofat least one of a size, position, and rotational orientation of thesensor object.
 15. The image synthesizing device of claim 14, whereinthe controller is configured to change, when the change of objectinformation of the sensor object is a change of position, a position ofthe designated object to correspond to the change of position of thesensor object.
 16. The image synthesizing device of claim 14, whereinthe controller is configured to change, when the change of objectinformation of the sensor object is a change of size, a size of thedesignated object by a size change ratio identical to the size changeratio of the sensor object.
 17. The image synthesizing device of claim14, wherein the controller is configured to change, when the change ofobject information of the sensor object is a change of rotationalorientation, a rotational orientation of the designated object tocorrespond to the change of rotational orientation of the sensor object.18. The image synthesizing device of claim 12, further comprising aradio frequency (RF) unit configured to transmit the synthesis image andthe changed synthesis image to another party's terminal when performingaudiovisual communication.
 19. The image synthesizing device of claim12, further comprising an input unit configured to select a stored imageas the designated image.
 20. The image synthesizing device of claim 12,further comprising an input unit including a sensor configured to detectan image input by a user in an image forming mode, wherein the imageformed by the user is designated as the designated image.
 21. The imagesynthesizing device of claim 20, wherein the sensor is at least one of atouch sensor, heat sensor, pressure sensor, terrestrial magnetismsensor, and acceleration sensor.
 22. The image synthesizing device ofclaim 20, wherein the display unit comprises a touch screen configuredto be used as the input unit.